Comparison of Czech Transformers on Text Classification Tasks

In this paper, we present our progress in pre-training monolingual Transformers for Czech and contribute to the research community by releasing models public. The need such emerged from effort employ language-specific tasks, but found performance of published multilingual be very limited. Since are usually pre-trained 100+ languages, most low-resourced languages (including Czech) under-represented these models. At same time, there is a huge amount training data available web archives like Common Crawl. We have publicly released two compared them with relevant public models, trained (at least partially) Czech. paper presents procedure as well comparison on text classification task various domains.